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FOREWORD 

This  report  may  be  of  most  value  to  the  relatively  inexperienced  human  factors  engineer 
who  must  support  the  analysis  activities  associated  with  a  systems  design  effort.  The  human  factors 
engineer  can  contribute  a  lot  to  such  an  effort,  and  ensure  that  sufficient  attention  is  paid  to  the 
operator  performance  part  of  the  analysis.  With  practice,  the  human  factors  engineer  can  do  most 
of  the  analysis,  providing  useful  results  to  the  rest  of  the  design  team;  the  results  can  also  be 
used  to  support  human  operator  requirements  with  hard  numbers. 

Systems  engineers  may  also  gain  an  understanding  from  this  report  of  how  operator  perfor¬ 
mance  considerations  can  be  included  in  systems  analysis  and  design. 

The  work  for  this  report  was  performed  at  the  Naval  Weapons  Center  (NWC),  China  Lake, 
Calif.  This  report  was  written  to  expand  on  a  section  of  NWC  TP  6541,  The  Human  Operator 
and  System  Effectiveness,  as  part  of  an  effort  by  the  U.  S.  Navy’s  Human  Factors  Block  program 
on  analysis  development  in  Human  Factors.  The  report  supercedes  the  NWC  TM  5332,  Measures 
of  Effectiveness  in  Systems  Analysis  (Including  Human  Factors).  The  work  was  supported  by  the 
Human  Factors  Block  under  the  direction  of  the  Naval  Air  Development  Center  (NADC), 
Warminister,  Penn.  The  work  was  conducted  between  January  and  July  1984  under  AIRTASK 
A330A330J/1001B/4WF57525000  and  in  June  1986  under  NADC  Work  Request  No.  N62269 / 
86/WR/0Q042,  NPRD  Document  No.  N68221/86/WR/60028. 
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EXECUTIVE  SUMMARY 


This  report  illustrates  that  the  MOEs  associated  with  any  particu¬ 
lar  system  or  analysis  are  hierarchical  in  nature.  The  top-level 
hierarchy  includes  factors  such  as  cost,  survivability,  reliability, 
and  capability.  The  second-level  hierarchy  includes  an  expansion  of 
each  of  these  MOEs.  The  development  of  capability  MOEs  is  the  topic 
of  this  report. 

The  procedures  to  follow  in  developing  capability  MOEs  tor  an 
effectiveness  analysis  include  separating  the  mission  objectives  from 
one  another,  identifying  the  functions  that  must  be  pertormed  to  com¬ 
plete  each  objective,  and  quantifying  the  performance  so  that  the  MOEs 
can  be  calculated.  This  report  gives  the  steps  and  techniques  to  fol¬ 
low  in  developing  MOEs.  Diagrams  showing  the  procedures  to  be  fol¬ 
lowed  are  given  at  the  end  of  the  report  (pages  40  through  45). 

The  performances  of  system  components,  or  of  the  human  operators 
of  a  system,  are  not  found  explicitly  at  all  in  the  top-level  hier¬ 
archy.  This  means  that  there  is  no  ’'visible"  connection  between  human 
performance  in  system  operation  and  the  overall  "worth”  of  the  system. 

In  the  "capability  hierarchy",  there  is  a  transition  between  mis¬ 
sion  MOEs  and  individual  task  performance.  It  is  possible  to  go  down 
at  least  two  levels  in  the  hierarchy  before  encountering  individual 
system  components  or  human  operators  of  components.  Hence,  again, 
there  need  not  be  a  "visible"  connection  between  mission  capability 
and  individual  human  operator  performance,  unless  special  attention  is 
given  to  making  the  connection  explicit. 
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INTRODUCTION 


The  analysis  of  the  operation  of  a  system  or  the  estimation  of  the 
effectiveness  of  a  system,  must  include  the  determination  of  measures 
of  effectiveness  (MOE).  Equivalent  terras  found  in  the  literature  are 
measures  of  merit,  figures  of  merit,  and  performance  criteria. 

MOEs  are  also  required  for  the  design  of  tests  and  human  factors 
experiments  (dependent  variables  are,  or  can  be  related  to,  MOEs). 
The  choice  of  relevant  MOEs  is  critical  to  the  validity,  credibility, 
and  acceptance  of  the  analysis  or  test  results. 

This  report  presents  information  from  a  number  of  sources  on  the 
development  of  MOEs.  Classification  schemes  are  provided  to  aid  in 
the  selection  of  MOEs  for  analysis  and  testing,  and  guidelines  are 
given  for  developing  MOEs  for  use  in  effectiveness  analysis. 


BACKGROUND 


The  general  procedures  to  be  followed  in  conducting  an  effective¬ 
ness  analysis  were  outlined  in  a  recent  report  (Reference  1).  One  of 
the  major  tasks  in  the  analysis  procedure  is  the  development  of  appro¬ 
priate  measures  of  effectiveness  (Box  3,  Figure  1).  Some  guidelines 
for  the  development  of  MOEs  are  given  in  Reference  1,  but  the  com¬ 
plexity  and  criticality  of  MOEs  were  not  given  appropriate  attention. 
This  report  expands  on  the  MOE  section  of  Reference  i. 


NOMENCLATURE  AND  MOE  CLASSIFICATIONS 


The  development  of  measures  of  effectiveness  is  a  broad  and  com¬ 
plex  topic;  MOEs  are  found  at  most  stages  of  research,  system  design, 
development,  and  evaluation,  although  they  are  not  always  called  MOEs. 


NWC  TP  6740 


This  report  is  intended  to  be  a  useful  guide  to  the  analyst  in 
developing  MOEs  for  use  at  a  particular  level  of  analysis.  It  is 
necessary  to  define  the  terms  and  limit  (or  delineate)  the  scope  of 
the  report  to  make  it  as  specific  and  useful  as  possible.  This 
section  of  the  report  will  aid  in  that  process. 


FIGURE  1.  Principal  Tasks  Required  for  Evaluation 
of  System  Effectiveness. 
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SYSTEM 

This  study  deals  with  systems  which  include  human  operators.  A 
definition  that  tits  the  topic  of  this  report  (and  the  analysis 
procedures  presented  in  Reference  1)  is  given  below. 

1.  A  man-machine  system  is  a  set  of  interfacing  components  com¬ 
posed  of  humans  and  machines  (including  software)  directed 
toward  performing  a  function  or  number  of  functions  and 
operating  within  the  constraints  of  time  and  specified 
environments. 


SYSTEMS  EFFECTIVENESS 

The  Air  Force  defines  systems  effectiveness  as  (Reference  2): 

l.  A  measure  of  the  extent  to  which  a  system  may  be  expected  to 
achieve  a  set  of  specific  mission  requirements,  and  which  may 
be  expressed  as  a  function  of: 

Availability 

Dependability 

Capability 

where , 

Availabilty  is  a  measure  of  the  condition  of  thf  system  at  the 
start  of  the  mission  at  any  point  in  time. 

Dependability  is  a  measure  of  the  system  condition  at  one  or 
more  points  during  the  mission. 

Capability  is  a  measure  of  the  ability  of  the  system  to  achieve 
the  mission  objectives  and  specifically  accounts  tor  the 
performance  spectrum  of  a  system. 

MOE  DEFINITIONS 

Component  measures  that  lead  to  the  quantification  of  system 
effectiveness  have  several  names: 

Measure  of  Effectiveness  (MOE) 

Figure  of  Merit  (FOM) 

Measure  of  Merit  (MOM) 

Effectiveness  Measure 
Effectiveness  Criterion 
Criterion  Measure 
Criterion 
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it  seems  to  occur  more  often  than  the  others  in  the  literature. 
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Reference  3  provides  a  good  definition  of  measure  of 
effectiveness : 

1.  An  MOE  generally  is  a  quantitative  expression  of  the  degree  to 
which  a  system  meets  its  objectives,  and  hence  an  analytical 
standard  of  comparison.  Also,  an  MOE  is  a  criterion 
expressing  the  extent  to  which  a  combat  system  performs  a  task 
assigned  to  Chat  system  under  a  specified  set  of  conditions. 

Suffice  it  to  say  that  there  are  a  number  of  definitions  of  MOE, 
and  that  they  are  all  dealing  with  the  same  or  very  similar  concepts. 
Further  definition  of  the  MOEs  in  this  report  can  be  made  by  way  of 
classification  schemes. 


MOE  HIERARCHY 
Level  of  the  Analysis 

There  are  several  levels  of  analysis,  ranging  from  that  conducted 
to  specify  component  characteristics  all  the  way  up  to  that  dealing 
with  the  large-scale  employment  of  a  number  of  systems.  The  hierarchy 
of  useful  analyses,  in  military  terms,  is  indicated  in  Table  1  -  with 
the  hastily  added  proviso  that  it  is  just  an  example  to  illustrate  the 
concept.  This  report  addresses  level  three  more  than  the  others. 

Figure  2  shows  the  concept  of  combining  MOEs  from  each  level  to 
get  to  the  next  higher  level.  This  report  deals  only  with  MOEs  for 
system  capability,  but  it  must  be  noted  that  these  MOEs  are  combined 
with  others  to  get  to  the  next  higher  level.  It  is  important  to  be 
aware  of  the  impending  combination,  so  that  whatever  is  generated  will 
be  usable  later. 

Figure  2  is  only  an  example  used  to  illustrate  the  concept  of  com¬ 
bining  MOFs.  Other  methods  of  combining  the  factors  shown  may  be  used 
in  some  analyses.  For  example,  the  survivability  MOE  may  be  an  input 
to  a  system  performance  MOE  in  some  cases. 

The  concept  of  building  up  to  a  system  capability  MOE  is  shown  in 
Figure  3.  The  performance  of  individual  components  or  human  operators 
is  combined  in  appropriate  fashion  to  obtain  the  desired  MOE.  Ways  of 
identifying  and  combining  these  performances  will  be  discussed  later. 
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TABLE  1.  Example  of  Analysis  Hierarchy. 


Level  of  analysis 

Footnotes 

7. 

Force-level  employment  of  many  systems 

C 

6. 

Tactical  employment  of  many  systems 

A,C 

5. 

Multi-element  mission  performance 

A,B 

4. 

Mission  performance  with  one  system 

A,B 

X 

3. 

Mission-segment  performance  with  system 

A,B 

X 

2. 

Subsystem  performance  (with  operators) 

A 

X 

1 . 

System  component  and  operator  performance 

A 

A  -  Should  include  operating  procedures  and  environmental 
conditions. 


B  -  Should  include  opposition  or  adversary,  if  applicable. 

This  may  result  in  additional  or  different  MOEs. 

C  -  Must  include  opposition  or  adversary,  if  applicable.  This 
may  result  in  additional  or  different  MOEs. 


Discussed  in  this  report 
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FIGURE  3.  Concept  ot  MOEs  Developed  from  Combining 
Component  Performances. 


Performance  Versus  Effectiveness 

The  terms  used  in  Figure  3  require  definitions;  for  example,  is 
"performance”  the  same  as  MOE?  When  does  one  merge  into  the  other? 
Some  guidelines  to  the  concepts  presented  in  this  report  are  given  in 
Table  2,  but  the  reader  should  be  aware  that  other  definitions  are 
used  and  lead  to  equally  valid  analysis  guidelines  and  results.  How¬ 
ever,  Table  2  will  be  useful  in  interpreting  the  concepts  in  this 
report . 
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TABLE  2.  Guideiines  to  Word  Usage  in  this  Report. 


Composite  MOE 


MOE  made  up  by  combining  other  MUEs. 


MOE 


A  quantitative  expression  of  the  degree  to 
which  a  system  meets  a  specific  objective. 


MOE  component 


Performance 


Any  quantity  that  is  used  in  Combination  with 
others  to  calculate  a  MOE;  a  quantity  calcu¬ 
lated  from  the  performance  values  (e.g.,  cur¬ 
sor  displacement  from  target)  and  other  para¬ 
meters  in  the  problem  (e.g.,  time)  to  produce 
a  capability  measure  (e.g.,  time-on-target). 

The  output  of  a  component,  human  operator, 
subsystem,  or  system. 


NOTE:  These  definitions  are  not  intended  to  be  mathematically 

rigorous,  mutually  exclusive,  or  the  only  ones  worth 
using.  They  are  intended  to  show  the  hierarchy  of  MOE 
formation. 


HUMAN  FACTORS  AND  MOEs 

Very  little  has  been  said  thus  far  about  the  connection  between 
human  factors  and  MOEs,  except  in  Figure  3.  Comments  made  about  human 
factors  testing  illustrate  chat  the  same  concepts  and  terms  used  in 
Table  2  have  been  in  the  human  factors  world  for  some  time 
(Reference  4): 

"Independent  variables  may  be  of  four  different  types: 

1.  Environmental  variables 

2.  Personnel  variables 

3.  Mission  variables 

4.  System  variables. 

Dependent  variables  are  measures  of  performance  outcomes  of  the 
system.  They  are  also  sometimes  called  criterion  measures  or 
more  simply,  criteria.  The  ideal,  or  ultimate,  criterion  is  one 
that  measures  the  performance  of  a  system  under  completely 
operational  conditions." 
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Meister  has  been  a  strong  advocate  of  a  systems  approach  in  human 
factors,  as  shown  in  the  edited  quote  below  (Reference  5). 

Personnel  subsystem  measurement  deals  with  performing  the 

total  task  or  job  in  the  context  of  the  actual  system  environ¬ 
ment.  The  meaningfulness  of  individual  and  team  performance 
data  lies  in  its  effect  on  the  higher  order  structure  (the 

subsystem  and  system)  in  which  performance  occurs. 

The  hierarchy  going  from  human  performance  dimensions  to  operator 
tasks  to  system  performance  measures  to  system  criteria  was  developed 
for  use  in  determining  future  test  development  in  space  applications 
(Reference  6).  The  relationships  between  mission  requirements,  per¬ 
formance  measures,  and  performance  criteria  were  also  discussed. 

There  are  other  explicit  examples  of  the  development  of  measures 
of  effectiveness  in  the  human  factors  literature.  Winterberg, 

Bricston,  and  Wulfeck  had  to  face  the  problem  of  developing  MOEs  for 

landing  aircraft  on  Navy  aircraft  carriers  (References  7  and  8). 
Ciavarelli  and  Williams  had  to  develop  MOEs  tor  air  combat  with  a 
training  application  (Reference  9).  And  Ketchei  and  McGrath  developed 
components  of  system  effectiveness  for  evaluating  the  performance  of 
airborne  forward  air  controllers  (Reference  1U). 

MOEs  have  been  included  in  human  tactors  work  for  some  time,  some¬ 
times  by  another  name.  This  report  will  attempt  to  synthesize  this 
and  other  work  to  produce  general  guidelines  for  developing  MOEs.  The 
link  between  operator  performance  and  the  measure  of  effectiveness 
will  be  illustrated. 

This  section  of  the  report  has  presented  the  concepts  of  a  system, 
system  effectiveness,  and  measures  of  system  effectiveness.  The  scope 
of  this  report  is  limited  to  developing  measures  of  effectiveness  for 
use  in  the  analysis  process  described  in  Reference  1;  that  is:  the 
capability  of  a  single  system  to  accomplish  its  mission  during  opera¬ 
tional  employment  at  the  mission  segment  level. 

It  is  important  to  note  that  this  report  addresses  only  a  small 
piece  of  the  total  picture,  as  shown  in  Figure  2.  The  other  factors 
shown  in  Figure  2  will  be  considered  in  most  programs,  but  their  dis¬ 
cussion  in  more  detail  is  outside  the  scope  of  this  report.  Those 
interested  in  other  MOE  development  (e.g.,  tor  training)  may  still 
find  some  useful  information  or  guidelines  in  this  report. 

Additional  supplemental  information  on  MOE  detinitions  and  desir¬ 
able  characteristics  is  given  in  the  Appendix.  For  those  with  limited 
background  in  this  aspect  of  analysis,  the  appendix  could  be  read  now. 
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MOE  DEVELOPMENT 


TOP-LEVEL  MOEs 

If  analysis  has  been  conducted  properly,  the  mission  has  been 
defined  and  the  system  has  been  described  (Figure  1).  These  activi¬ 
ties  shall  be  reiterated  in  the  following  text,  with  a  focus  on 
developing  specific  capability  MOEs. 


Separate  Missions 

Most  systems  are  required  to  perform  more  than  one  assignment  or 
mission.  It  is  necessary  to  keep  these  missions  separate  in  analysis 
and  associated  discussions,  or  considerable  confusion  can  result. 
Figure  4  shows  the  first  cut  at  breaking  the  general  mission  objec¬ 
tives  into  units  that  are  appropriate  for  analysis.  Figure  5  shows 
two  examples  of  this  process.  In  the  automobile  example,  only  one 
separation  is  required,  whereas  the  aircraft  example  illustrates  that 
sometimes  separation  should  be  taken  to  a  second  level. 

The  reader  should  note  that  the  procedures  figures  (e.g.  , 
Figure  4)  given  throughout  this  section  of  the  report  are  reproduced 
all  together  at  the  end  of  the  report  for  more  convenient  reference 
and  use. 

The  single  "concept  of  employment"  can  be  used  as  a  check  to  see 
that  there  are  only  one  system  and  one  associated  mission  objective 
being  listed  at  a  time. 


Define  General  Functions 

A  further  test  of  whether  or  not  the  missions  have  been  separated 
properly  is  to  compare  the  functions  that  must  be  performed  in  each 
mission.  The  functions  therefore  must  be  defined  at  this  point  in  the 
analysis. 


Figure  6  shows  that  the  system  description,  mission  objective,  and 
concept  of  employment  are  all  used  to  derive  the  general  functions 
that  must  be  performed  to  complete  the  mission  as  conceived.  This 

identification  of  functions  should  also  be  conducted  at  a  top  level, 
without  getting  too  specific.  A  diagram  similar  to  Figure  7  can  be 
used  to  illustrate  the  functions  arranged  along  a  time  or  distance 

line.  Important  events  can  be  shown,  or  the  mission  timeline  could  be 

broken  into  segments  if  it  is  helpful  to  the  analyst. 
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A  PROVIDE  DAILY  8  PROVIDE 

TRANSPORTATION  TRANSPORTATION 

FOR  PERSONNEL  FOR  FAMILY 

(HOME-WORK-SCHOOL)  VACATION  (PERSONNEL. 

CAMPING  GEAR) 


C  PROVIDE 

TRANSPORTATION 
FOR  ONE  PERSON 
FOR  AMATEUR 
RACING 
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Functions  should  be  defined  lor  each  of  the  separate  missions. 
These  Lists  or  diagrams  of  functions  can  then  be  compared  across  the 
separate  missions;  if  they  are  different,  the  missions  have  been 
properly  separated.  If  lists  of  diagrams  are  the  same,  the  missions 
from  which  they  were  derived  might  be  combined  again. 
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FIGUKt  6.  Inputs  to  the  Development  of 
General  Functions  tor  System  Capability. 
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a.  Timeline  with  Events  Shown. 
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b.  Functions  Shown  Along  Timeline. 


FIGURE  7.  Diagram  of  Events  and  General  Functions 
That  Must  Be  Performed  During  Mission. 


Note  that  a  function  allocation  (a  human  factors  analysis  process) 
has  not  yet  been  done.  The  missions  have  been  separated  to  the  point 
where  their  top-level  functions  differ.  Top-level  MOEs  can  now  be 
generated. 

Example  functions  for  the  two  systems  shown  in  Figure  5  are  given 
in  Tables  3  and  4.  The  functions  for  daily  and  vacation  transpor¬ 
tation  are  very  similar  in  Table  3.  The  analyst  might  combine  these 
two  "missions"  at  this  point  in  the  analysis.  The  amateur  racing 
function  are  clearly  different  from  the  others,  so  racing  must  be  kept 
as  a  separate  mission. 
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TAbLK  i.  General  Functions  to  be  Performed 
with/by  a  Transportation  System. 
(Assume  system  is  an  automobile.) 


A.  DAILY  TRANSPORTATION 

1.  Unlock  automobile 

2.  Load  automobile  (people) 

1.  Start  automobile 

4.  Drive  automobile 
Knter  trattic 

Follow  public  roadways  (paved) 

3.  Park  automobile 
b.  Unload  automobile 

7.  Lock  automobile 

b.  family  vacation  transportation 

1.  Unlock  automobile 

2.  Load  automobile  (people  and  camping  equipment) 
1.  Start  automobile 

4.  Drive  automobile 

Knter  trattic 

Follow  public  roadways  (paved  and  gravel) 

5.  Park  automobile 

b .  Unload  automobile  (people  and  equipment) 

7.  Lock  automobile 

8.  Conduct  day  trips  (as  in  Part  A) 

C.  AMATKUR  RACING 

1.  Load  automobile  (one  person) 

2.  Drive  automobile  onto  trailer 

3.  Unload  automobile 

4.  Tie  down  automobile  on  trailer 

5.  Tow  automobile  to  raceway 

b.  Drive/push  automobile  off  of  trailer 

7.  Fuel/service  automobile 

8.  Load  automobile  (one  person) 

9.  Start  automobile 

10.  Drive  automobile  on  raceway 

11.  Drive  automobile  onto  trailer 

12.  Unload  automobile  (one  person) 

13.  Tie  down  automobile  on  trailer 

14.  Tow  auto  home 
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TABLE  4.  General  Functions  to  be  Performed  with/by 
a  Target  Acquisition  System. 

A.  SEARCH  &  RESCUE 

1.  Activate  system 

2.  Test  system  functions 

3.  Select  target  signatures  to  be  detected 
(e.g.,  beacon  characteristics) 

4.  Fly  specific  pattern/route 

5.  Search 

Scan 

Detect  objects 
Sort/identify  objects 

6.  Identify  pre-designated  "target" 

7.  Determine/ record/mark,  target  location 

B .  ATTACK 


-s 


1.  Activate  system 

2.  Test  system  functions 

3.  Fly  to  target  area  as  preplanned 

4.  Search 

Scan 

Detect  objects  in  scene 
Maintain  geographic  orientation 
Identify  target  cues 
Detect/identify  target 

5.  Track  target 

C.  SURVEILLANCE/RECONNAISSANCE 

Activate  System 
Test  system  functions 
Fly  specific  pattern/route 
Search 

Detect  objects 
Record  images 

Record  sensor  location  and  pointing  angles 
Identify  potential  targets 
Return  to  base 
Play  back  or  view  imagery 
Reduce/ interpret  images  and  locations 


1. 

2. 

3. 

4. 


5. 

6. 

7. 
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Each  top-level  MOE  should  have  three  parts: 

1.  A  Statement  ot  Capability 

2.  A  Statement  of  Operating  Conditions 

3.  General  Quantitication  "Rules"  or  Concepts 

The  capability  and  operating  conditions  are  derived  from  the  mission 
objective  and  employment  concepts.  The  quantification  concept  is 
required  in  the  generation  of  the  components  of  the  top-level  MOE. 

Note  that  this  preliminary  set  of  MOEs  (or  perhaps  only  one  MOE ) 
should  be  related  only  to  the  mission  objective,  not  to  the  mission 
tunctions  shown  in  figure  7. 

Develop  Top-Level  MDE 

Table  5  lists  the  groups  associated  with  an  effectiveness 
analysis  including  engineers,  managers,  and  sponsors.  A  formal  effort 
should  be  made  to  get  from  these  people  the  questions  they  would  like 
to  have  answered  by  the  analysis. 


TABLE  5.  Example  of  the  Participants  in  the  System 
Effectiveness  Analysis  Process. 


Participants 

Example  ot  specialty 

Remarks 

L.  Analysts 

Systems  Analysis 

Human  factors 

Members  ot  concept, 
design,  and  evalua¬ 
tion  team. 

2.  Engineers 

Mechanical 

Electronic 

Optical 

Human  factors 

Principal  members  of 
the  design  team. 

3.  Scientists 

Atmospheric 

Physics 

Compute  r 

Usually  consultants. 

4.  Current  users 

Pilot 

Vehicle  Driver 

1 

May  not  be  near  design 
team,  or  understand 
the  design  process. 

b.  Managers 

Branch  Chief 

Project  Head 

Program  Manager 

i _ L 

Coordinate  and  direct 

work. 

TABLE  3.  (Coned.) 


Participants 

Example  of  specialty 

Remarks 

6.  Principal 
design  team 
members 

i 

Managers 

Analysts 

Engineers 

User  representatives 

Should  be  a  cohesive 
group  each  with  de¬ 
signated  areas/ 
responsibilities . 

7.  Sponsors 

i 

[ 

Higher  Level  Manager 

Can  be  regarded  as 
customer,  but  may  not 
be  the  actual  user. 

8.  Administration 

Budget 

Procurement 

l 

May  only  be  concerned 
the  procurement, 
schedules,  and  the 
budget  process. 

At  this  point,  then,  the  analyst  has:  (1)  a  single-mission 
objective,  (2)  a  top-level  system  description,  (3)  concepts  ot 
employment  of  the  system,  and  (4)  some  of  the  questions  that  need 
answering.  Table  6  was  derived  from  guidelines  given  in  Reference  11, 
pages  3-13,  and  can  be  used  in  developing  a  preliminary  set  ot 
top-level  MOEs.  Table  7  is  a  similar  set  of  instructions  for  this 
same  process  (Reference  12);  Tables  6  and  7  are  combined  in  Table  8  to 
give  this  report's  recommendations  in  developing  a  top-level  MOE. 
Table  8  is  reproduced  at  the  end  of  the  report  together  with  the 
procedures  diagrams  for  more  convenient  access  by  the  user. 


TABLE  6.  Steps  in  MOE  Development  (Reference  11). 


Question:  What  concepts  can  be  used  to  estimate  whether  or  not 
a  mission  objective  has  been  reached? 

1.  Create  as  many  MOEs  as  possible;  brainstorm,  even  though 
at  first  many  of  them  may  appear  to  be  alike. 

2.  Categorize  these  MOEs  into  groups  of  similar  measures. 

3.  Classify  the  MOEs  as  being  strong  or  weak. 

4.  Eliminate  MOEs  from  the  list  because  of: 

a.  Technical  inf easibility 

b.  Economic  inf easibi lity 

c.  Alternatives  clearly  dominate  MOEs 
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TABLE  7.  More  Steps  in  MOE  Development  (.Reference  12) 

List  the  important  mission  features,  so  that  the  MOEs 
will  have  a  better  chance  of  reflecting  the  way  a  mis 
sion  must  be  conducted  to  be  effective. 

Develop  an  extensive  list  of  conceivable  MOEs  tor  the 
mission,  without  initial  constraints  on  this 
brainstorming. 

Reduce  this  list  by  discarding  duplication  and  MOEs 
that  are  not  in  some  way  related  to  the  mission 
objective. 

Write  a  brief  discussion  of  each  of  the  MOEs  and  give 
the  analyst's  views  of  some  of  the  general  charac¬ 
teristics  of  each  MOE.  Include: 

a.  Relation  of  MOE  to  mission  objective 

b.  Inherent  assumptions  connected  with  MOE 

c.  Ways  the  MOE  can  be  misinterpreted  or  misleading 
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TABLE  B#  Steps  to  Follow  in  Thinking  up  Candidates 
tor  Top-Level  MOEs. 

1.  List  the  single  mission  objective  as  determined  in  the 
procedure  illustrated  in  Figure  4. 

2.  List  questions  that  need  answering  as  obtained  from 
sponsors,  program  managers,  design  engineers,  etc. 

3.  Develop  a  list  of  many  conceivable  MOEs  for  the  mission 
objective,  without  any  initial  constraints.  Use  a 
brainstorming  technique  if  possible. 

4.  Write  a  description  of  the  quantification  concepts  tor 
each  MOE  listed. 

5.  Categorize  the  MOEs  into  groups  of  similar  measures. 

6.  Reduce  the  list  by  combining  (or  discarding)  duplicate  or 
redundant  MOEs. 

7.  Further  eliminate  MOEs  from  the  list  because  of: 

a.  Technical  infeasibility 

b.  Economic  infeasibility 

c.  Weak  or  no  relation  to  mission  objective 

B.  List  the  remaining  top-level  MOEs  tor  the  single  mission 
objective  given  in  (I)  above. 

9.  Repeat  the  process  for  each  single  mission  objective. 


24 


V  P  vu 


IF*  'lFM%pn.T  '  VI VJW 


1 


NWC  TP  6740 

Two  examples  of  using  Table  8  on  missions,  taken  from  Figure  5, 
are  given  in  Tables  9  and  10. 


TABLE  9.  Example  of  Top-Level  MOE  Process  for  Family  Vacation 
Transportation.  (According  to  Table  8.) 


1.  Objective:  Provide  transportation  for  single  family  and  camping 
gear  during  vacation. 

2.  Questions:  How  many  people  must  vehicle  transport?  How  much 
camping  gear  must  be  hauled?  (Sample  tor  this  example.) 

3.  List  of  MOEs 

a.  Mileage 

b.  Number  of  passengers  possible 

c.  Volume  of  cargo  space. 

d.  Total  volume  for  passengers  and  gear 

e.  Cargo  capacity  (weight) 

f.  Distance  between  refueling 

g.  Total  hauling  weight  (passengers  plus  cargo) 

h.  Speed  up  5%  incline  with  full  load 

i.  Number  adults  and  number  children 

j .  Comfort  of  passengers 

4.  Quantification  of  (3)  above 

a.  Miles  per  gallon 

b.  Number  of  seats 

c.  Cubic  feet,  size  of  rectangular  box 

d.  Cubic  feet  that  would  fit  in  space 

e.  Pounds 

f.  Miles 

g.  Pounds 

h.  Miles  per  hour 

i.  Number  of  large  seats  and  number  of  small  seats 

j.  Comfort  rating  from  testing;  volume  per  passenger 

5.  Combine  items  in  (3)  above 

a.  Mileage  and  distance  between  refueling 

b.  Total  hauling  weight  =  cargo  capacity  plus  passenger  weight. 

c.  Volume  remaining  after  passengers  loaded  (total  volume  -  seat 
and  passenger  volume) 
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TABLE  9.  (Contd.) 

6.  Reduce  MOE  list 

a.  Delete  mileage;  it  is  cost  item  rather  than  capability  item. 

b.  Delete  number  of  adults  and  children;  consider  all  passengers 
equally. 

c.  Delete  total  hauling  weight;  it  duplicates  number  of  passen¬ 

gers  and  cargo  weight. 

d.  Delete  total  volume;  it  duplicates  number  of  passengers  and 

cargo  volume  . 

7.  Further  reduce  MOE  list 

a.  Delete  comfort  index.  It  is  too  difficult  to  handle  in  paper 
analysis  only. 

8.  Kemaining  MOEs 

a.  Number  of  passengers  possible 

b.  Cargo  volume  remaining  after  passengers  loaded 

c.  Cargo  weight  capacity 

e.  Distance  between  refueling 

f.  Speed  up  5%  grade 
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TABLE  10.  Example  of  Top-Level  Process  tor 
Air-to-Ground  Target  Acquisition  System  for  Attack. 
(According  to  Table  8.) 


1.  Objective:  Acquire  targets  for  attack 

2.  Questions:  What  sensors  should  be  used  in  the  system?  Can  we 

use  stand-off  weapons  with  the  system?  How  will  system  employ¬ 
ment  affect  survivability? 


3.  List  of  MOEs 

a.  Range  of  target  acquisition 

b.  Targets  acquired 

c.  Probability  of  acquiring  targets 

d.  Search  time  required 

e.  False  alarms 

f.  Times  system  is  severely  degraded  by  weather 

g.  Type  of  targets  against  which  system  is  effective 

4.  Quantification  of  (3)  above 

a.  Thousands  of  feet 

b.  Percent  of  total  available 

c.  Cumulative  percent  versus  range  in  thousands  of  feet 

d.  Seconds  under  fixed  geometric  conditions  (same  altitude  and 
range) 

e.  Percent  of  total  reports  that  are  not  real  targets 

f.  Percent  hours  per  day;  percent  days  per  year;  percent  per 
month  as  function  of  month  (NOTE:  Geographic  location  must 
be  specified) 

g.  Moving  targets;  radar-reflective  targets,  hot  (infrared) 
targets;  large  targets;  camouflaged  targets 

5.  Combine  items  in  (3)  above: 


6. 


7. 


a.  (a),  (b),  and  (c)  are  similar 


Reduce  MOE  List: 
a.  Delete  (a)  and  (b); 


use 


1 

1 

1 

I 


(c) 


1 

i 


Further  reduce  MOE  List 


a.  Delete  search  time  required  since  it  is  implied  in  (c)  as 
range  to  target  closes. 
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TABLE  10.  (Contd.) 


8.  Remaining  MOEs: 

a.  Cumulative  probability  of  acquiring  targets  as  a  function 
of  range 

b.  Percent  false  alarms 

c.  Percent  weather  degradation 

d.  Applicable  target  types 


NOTE:  Acquisition  range  could  relate  to  survivability  and  stand¬ 

off  weapons;  sensor  types  relate  to  target  types.  MOEs  do 
not  explicitely  address  all  questions  in  this  example. 


MOE  Clarification 

If  the  mission  objective  has  been  described  very  specifically  in 
great  detail,  the  first  definition  of  the  MOE  may  be  good  enough;  this 
is  not  usually  the  case,  however.  It  is  usually  necessary  to  clarify 
the  mission  objective  so  that  the  MOE,  in  turn,  can  be  made  specitic 
enough  to  quantify.  This  process  can  involve  iteration,  negotiation, 
and  compromise  among  the  managers,  sponsors,  and  design  team  members. 
It  may  occur  several  times  throughout  the  design  phase. 

The  clarification  is  often  a  simplification  of  the  "super"  cap¬ 
abilities  initially  desired  by  most  sponsors  and  design  engineers. 
The  three  descriptors  of  a  MOE  (capability,  operating  conditions,  and 
quantification  concepts)  can  be  specified  or  simplified  more  or  less 
independently,  although  sometimes  they  are  interrelated.  Some 
examples  of  the  stages  of  clarification  are  shown  tor  four  systems  in 
Tables  11,  12,  and  13. 


Top-Level  MOE  Approval 

After  following  the  guidelines  provided  in  Table  8,  and  clarifying 
the  MOE,  there  should  be  three  descriptors  (capability,  operating  con¬ 
ditions,  and  quantification  concept)  for  each  MOE.  There  will  be  one 
or  more  MOE  for  each  separate  mission.  These  top-level  MOEs  should  be 
approved  by:  (1)  sponsor,  (2)  project  or  program  manager,  and  (3) 
other  design  team  members.  Potential  or  current  system  users  should 
also  be  asked  if  the  MOEs  mean  anything  to  them  in  the  operational 
world.  Do  the  MOEs  address  the  questions  that  people  are  asking? 
Will  the  MOEs  provide  the  data  needed  to  aid  in  decision  making? 

Another  iteration  may  be  necessary  to  obtain  a  consensus  on  ail 
aspects  of  the  MOEs. 
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TABLE  1L.  Examples  of  Che  Stages  of  Specifying  (Usually 
Reducing)  a  System's  Desired  Performance. 


System 

Stage 

1  -  i 

Performance 

1.  Antitank  weapon 

1st 

Destroy  tanks 

2nd 

Destroy  mobility  of  tanks 

3rd 

Degrade  weapon  accuracy 
(firepower)  of  tanks 

2.  Antiship  missile 

1st 

Sink  any  ship 

2nd 

Destroy  defensive  fire 
power  on  any  ship 

3rd 

Destroy  radar  on  any  ship 

4  th 

Destroy  defensive  fire¬ 
power  on  a  particular 
(enemy)  ship 

3.  Fly  fishing  outfit 

1st 

Catch  all  kinds  of  fish 

2nd 

Catch  only  trout 

3rd 

Catch  trout  on  dry  flies 
only 

4.  Pilot  ejection/recovery 

1st 

Save  pilot  completely  un- 

system  (aircraft 

harmed 

subsystem) 

2nd 

Save  pilot;  temporary 
spinal  disk  compression 
permissible 

3rd 

Save  pilot;  some  temporary 
injury  permissible 
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TABLE  12.  Examples  of  Stages  of  Specifying  Operating  Conditions. 


System 

Operating  Conditions 

1.  Antitank  weapon 

1st 

Anywhere,  anytime 

2nd 

On  or  near  roads;  fair  to  good 
weather;  night  and  daytime 

3rd 

On  or  near  roads;  fair  to  good 

weather;  daytime 

2.  Antiship  missile 

!  1st 

Anytime,  anywhere 

:  2nd 

Latitudes  +  80°;  ail  types  of 
weather;  day  and  night 

3rd 

Latitudes  +  80°;  fair  to  good 
weather;  day  and  night 

3.  Fly  fishing  outfit 

1  st 

All  waters,  daytime,  all  sea¬ 
sons 

2nd 

Streams,  rivers,  and  lakes, 

1  daytime,  summer 

3rd 

Streams  and  rivers,  daytime, 
summer 

4.  Pilot  ejection/ 

1st 

Whenever  pilot  is  in  the  air- 

recovery  outfit 

craft 

2nd 

Anywhere;  aircraft  speed  >50 

1 

knots;  aircraft  in  upright 
position 

3rd 

- 

200  ft  <  altitude 
<  60,000  ft; 
speed  >  30  knots 

t 

> 

I 

i 

I 

I 


\ 
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TABLE  13.  Examples  ot  Stages  in  the  Development  ot 
MOE  Quantification  Concepts. 


System 

Stage 

Quantification  Concept 

1.  Antitank  weapon 

1st 

Probability  of  destroying 
tanks  on  one  mission.  Number 
ot  tanks  that  can  be  destroyed 
by  one  system  on  one  mission. 

2nd 

Percent  tanks  destroyed  by  one 
system  on  one  engagement. 

l  3rd 

Percent  tanks  ot  a  total  of 
nine  tanks  moving  along  a  road 
that  can  be  "mobility-killed" 
on  one  pass  through  the  area 
by  one  system. 

2.  Antiship  missile 

1st  , 

Probability  of  sinking  a  ship, 
given  missile  launch. 

2nd 

Probability  of  hitting  a  ship, 
given  missile  launch. 

3rd 

i 

Probability  of  hitting  an 
enemy  ship,  given  a  missile 
launch. 

4th 

Probability  of  destroying  the 
defensive  firepower  on  a  par¬ 
ticular  enemy  ship,  given  mis¬ 
sile  launch. 

3.  Ely  fishing  outfit 

l  St 

Number  ot  fish  caught  in  a  day 

2nd 

Number  of  trout  hooked  on  a 
dry  fly  in  one  day. 

3rd 

Number  of  trout  hooked  on  a 
dry  fly  along  a  given  stretch 
of  river  between  0900  and  1200 
on  a  sunny  day. 

4.  Pilot  ejection/ 

1st 

Probability  of  safely  eject 

recovery  system 

i 

ing/ recover ing  a  pilot  when 
called  upon  to  do  so. 

2nd 

Probability  that  ejection  is 
within  specifications;  e.g., 
acceleration,  duration,  motion 
envelopes,  etc. 

3rd 

Probability  that  all  subsys¬ 
tems  function  within  specifi- 

_ i 

cations. 
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SECOND-LEVEL  HOEs 

The  general  functions  (Figure  7,  Tables  3  and  4)  have  only  been 
used  thus  far  to  determine  when  to  stop  separating  missions.  They  can 
now  be  used  to  generate  the  second-level  MOEs. 

It  is  important  to  note  that  the  general  functions  have  not  yet 
been  allocated  to  any  particular  subsystem,  component,  or  human 
operator.  However,  MOEs  usually  can  still  be  developed  tor  each. 


General  Function  Quantification 

The  general  functions  can  be  described  in  terms  of  the  system, 
mission,  and  employment  concept.  The  descriptions  should  provide 
enough  detail  for  use  in  listing  ways  that  performance  of  the  func¬ 
tions  might  be  quantified.  The  processes  given  in  Table  3  can  be  used 
to  develop  alternative  quantification  methods  similar  to  those  shown 
in  Table  13.  Table  14  gives  an  example  taken  from  the  antitank  weapon 
system  mentioned  earlier.  Table  15  is  included  to  illustrate  that  the 
same  techniques  can  be  used  on  vastly  different  systems  (tank  attack 
and  fly  fishing).  Besides,  it's  fun'  to  think  about. 

Table  14  lists  more  than  one  quantification  concept  for  any  one 
function;  the  same  could  have  been  done  tor  Table  15.  It  is  necessary 
to  select  one  concept  from  the  alternates  for  use  in  calculating  the 
top-level  MOE.  The  concept  that  is  selected  should  be: 

1.  Required  to  calculate  the  top-level  MUE  that  has  already 
been  selected 

2.  Related  to  the  questions  to  be  answered  in  the  analysis 

3.  Based  on  system  characteristics  and  employment  concepts 
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TABLE  14.  Brief  Example  of  General  Function  Listing 
and  Performance  Quantification  Concepts. 


System:  Antitank  Weapon 
General  Functions: 

a.  Arrive  in  target  area 

b.  Search  for  target 

c.  Acquire  target 

d.  Fly  to  target 

e.  Damage  target 

Performance  Quantification  Concepts  (Second  level  MOEs): 

a.  Accuracy  in  arriving  at  predesignated  point;  heading 
accuracy  when  entering  target  area;  time  accuracy  compared 
to  required  " time-on-target" . 

b.  Time  spent  in  target  area  before  target  is  found;  number  of 
passes  through  target  area  before  target  is  tound;  range  at 
which  target  is  tound. 

c.  Percent  targets  found;  number  of  engagements  when  non-target 
is  identified  as  a  target;  cumulataive  percent  targets  tound 
as  a  function  of  range  from  the  target. 

d.  Miss  distance  (in  two  or  three  dimensions). 

e.  Probability  of:  Completely  destroying  the  target  and  crew; 
destroying  the  mobility  of  the  target;  damaging  the  target 
so  it  cannot  perform  as  required  in  combat. 
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TABLE  15.  Another  Brief  Example  of  General  Function  Listing 
and  Performance  Quantification  Concepts. 

System:  Fly-Fishing  Out  tit 

General  Functions: 

a.  Identity  likely  location  of  trout  in  stream  and/or 

b.  Locate  any  trout  rising  to  feed  on  the  surface 

c.  Select  dry  tly  and  tie  onto  flyiine 

d.  Move  (includes  wading)  into  casting  position 

e.  Cast  fly  to  correct  location 

f.  Hook  trout  wben/it  strike  occurs 

g.  Play  trout  to  where  it  can  be  netted 

h.  Net  trout 

Performance  Quantification  Concepts  (Second  level  MOEs): 

a.  Percent  "likely  locations"  correctly  identified  along 
given  stretch  of  a  stream/river. 

b.  Percent  of  rising  trout  correctly  spotted. 

c.  Probability  of  selecting  "reasonable"  fly  tor  the  occa¬ 
sion;  percent  of  flies  correctly  tied  onto  line. 

d.  Percent  of  the  time  good  casting  position  can  be  taken 
without  scaring  trout;  percent  of  the  time  position  can  be 
taken  without  falling  or  getting  wet  (wader  evaluation). 

e.  Percent  of  the  time  fly  is  cast  to  the  proper  location  and 
lands  without  scaring  fish. 


f. 

Percent 

hooked. 

of 

Che 

strikes 

during 

which  the 

trout  is  also 

g  6.  h. 

Percent 

until  it 

of 

can 

the 

be 

time  the 
netted. 

trout 

is  played 

and  brought  in 

Combining  Second-Level  MOEs 

As  has  been  implied  above,  a  second-level  MOE  is  defined  as  a  mea¬ 
sure  of  how  well  a  general  function  is  performed.  When  the  general 
functions  are  all  performed  as  planned,  the  mission  has  also  been  per¬ 
formed,  or  completed. 


The  top-level  MOE  is  calculated  from  the  second-level  MOEs,  with 
care  taken  to  correctly  combine  appropriate  metrics.  This  is  the 
point  that  mathematics  enters  the  MOE  development  process,  and  to  some 
extent,  the  development  of  the  effectiveness  algorithm  (Figure  1)  has 
begun. 
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The  way  that  the  second-level  MOEs  are  combined  is  a  function  of 
their  particular  characteristics,  and  the  particular  problem  at  hand. 
Developing  several  methods,  or  listing  many  examples  from  "case 
studies"  are  beyond  the  scope  of  this  paper;  one  example  will  be 
given,  however. 

Probability/Percent.  One  MOE  is  the  probability  of  completing  a 
mission,  and/or  percent  missions  successfully  completed.  In  other 
words,  the  expectation  one  might  have  of  success,  and  the  fraction  of 
the  time  that  things  will  proceed  as  desired. 

Table  14  gives  brief  examples  of  the  general  functions  listing  and 
the  concepts  of  performance  quantification.  The  following  paragraphs 
give  a  better  understanding  of: 

L.  The  measures  in  Table  14  could  be  stated  as: 

a.  Probability  of  arriving  in  the  target  area  within  +1  mile 
in  range  and  degrees  in  heading  from  that  planned. 

b.  Probability  of  getting  the  targets  within  view  of  the 
system  sensors. 

c.  Probability  of  detecting  and  recognizing  the  target  in 
time  to  turn  toward  it. 

d.  Probability  of  coming  within  10  feet  of  the  target. 

e.  Probability  of  damaging  the  target  to  the  desired  level 
(see  top  level  MOE). 

The  top-level  MOE  could  then  be  expressed  as: 

^success  =  ^a  x  ^b  x  ^*c  x  ^d  x  ^e  ( ^ ) 


The  probabilities  are  conditional;  that  is,  the  system  could  not  get 
within  10  feet  of  the  target  unless  it  had  arrived  in  the  target  area. 

The  probabilities  can  be  simply  combined  as  shown  above  if  they 

are  independent,  a  condition  that  must  be  established.  If  they  are 

not  independent,  a  different  formulation  would  have  to  be  developed. 

Redundant  (or  parallel)  data  processing  and  actions  would  also  lead  to 
a  different  formulation  than  the  "series"  situation  represented  in 
Equation  (1). 

It  can  be  seen  that  the  probabilities  listed  in  the  example  are 
getting  specific  enough  so  that  the  employment  concepts  and  system 
characteristics  are  pertinent  in  their  formulation.  The  next  step  in 
the  process  is  to  figure  out  how  to  calculate  each  of  the 

probabilities. 
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THIRD-LEVEL  MOEs 

In  the  procedure  that  has  been  developed  thus  far  in  this  report, 
some,  or  most  third-level  MOEs  could  be  considered  to  be  measures  of 
performance,  as  indicated  in  Figure  3. 

It  is  not  worth  quibbling  about  whether  to  call  these  quantities 
measures  of  effectiveness  or  measures  of  performance.  It  helps,  of 
course,  if  any  given  technical  report  defines  its  terms  and  is  consis¬ 
tent  in  their  use.  Suffice  it  to  say  that  one  often  finds  a  tran¬ 
sition  in  terminology  between  measures  of  effectiveness  of  a  mission, 
and  measures  of  performance  of  a  single  task  performed  in  the  conduct 
of  the  mission,  sometimes  with  undefined  terms  separating  the  two 
extremes. 


Human  Operator  Performance 

The  capability  of  the  human  operator  to  perform  tasks  in  system 
operation  enters  the  analysis  at  about  the  third  levei  in  the  MOE 
hierarchy  (Figure  3).  In  many  cases,  human  performance  is  a  direct 
input  to  calculating  one  or  more  of  the  probabilities  shown  in 
Equation  (1). 

Probability-type  MOEs,  like  those  in  the  example,  require  prob¬ 
ability-type  input  data;  the  form  of  operator  performance  data 
discussed  in  Reference  1  stresses  this  point.  Operator  performance 
data  to  be  used  in  analysis  must  be  in  a  particular  form  (as 
determined  by  the  analysis).  For  example,  mean  scores  will  not  be 
useful  if  score  distributions  are  required.  It  follows  that  the 
people  generating  human  performance  data  must  either  have  exact  data 
requirements  (or  specifications),  or  themselves  be  familiar  with  the 
nature  of  analysis. 


Functions  and  Tasks 

The  functions  illustrated  in  Figure  7  must  be  made  more  specific 
to  develop  the  third-level  MOEs.  In  some  cases,  several  tasks  must  be 
performed  to  accomplish  a  function;  the  functic  i  MOE  would  then  be 
made  up  of  several  measures  of  task  performance,  combined  in  a 
physically  and  mathematically  appropriate  way. 

One  example  from  Tables  11,  13,  and  14  is  summarized  in  Table  16. 
The  second-level  MOEs  are  expanded  into  functions,  as  mentioned 
previously,  in  Table  17. 
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TABLE  16.  Example  of  Top-  and  Second-Level  MOEs 
tor  Antitank.  System. 


MOEs 

Quantification 

Top- level 

Ability  to  destroy 

Percent  tanks  of  a  total  of  nine  tanks 

mobility  of  tanks 

moving  aiong  a  road  that  can  be 

"mobility-killed"  on  one  pass  through 

the  area  by  one  system. 

Second-level 

1.  Ability  to 

Cumulative  percent  targets  found  as  a 

acquire  target 

function  ot  range  from  the  target. 

2.  Ability  to 

Cumulative  percent  of  aircraft  that  find 

fly  toward 

target  in  time  to  fly  to  correct  weapon 

target 

release  point. 

3.  Ability  to 

Miss  distance  (in  two  or  three 

damage  target 

dimensions)  of  the  weapon.  Probability 

of  destroying  the  mobility  of  the 

target. 

TABLE  17.  Expansion  of  Second-Level  MOEs 
Shown  in  Table  16. 


Second-level 

MOEs 

Functions 

Measure  of 
performance 

1.  Ability 
acquire 

to 

target 

a. 

Scan  terrain 
ahead  of  aircraft 

Probability  that  tar¬ 
get  is  brought  within 
t ield-of-view  (FOV). 

b. 

Detect  target 

Probability  that  tar¬ 
get  is  detected  given 
that  it  is  in  FOV. 

c. 

Identify 

target 

i 

1 

Probability  that  tar¬ 
get  is  identified  as 
something  to  be  at¬ 
tacked,  given  that 

it  is  detected. 
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TABLE  17.  (Contd.) 


Second-  level 

MOEs 

Funct ions 

Measure  ot 
pe  rf ormance 

2.  Ability  to  fly 
toward  target 

a.  Turn  aircraft 

Koli-in  time 
"G"s  in  turn 

o.  Roll  out  ot  turn 

1  | 

I 

Time 

Accuracy  in  finai 
heading 

c.  Fly  toward  target 

' 

Tracking  accuracy 
Tracking  time  re¬ 
quired 

Probability  of  flying 
to  correct  weapon 
release  points. 

3.  Ability  to 
damage  target 

a.  Weapon  flies  to 
target 

Miss  distance 

b.  Warhead  detonates 

Probability  ot 
detonating 

c.  Explosion/ t rag- 
mentation  damages 
target 

Level  ot  damage 
given  miss  distance 

AC  this  level  in  the  analysis,  a  function  allocation  will  have 
been  done,  where  the  functions  are  "assigned”  to  system  components  or 
human  operators.  Performance  requirements  tor  both  humans  and  com¬ 
ponents  can  be  established.  At  this  point  in  the  analysis,  the 
analyst  has  really  moved  out  ot  Box  3  in  Figure  1  (.specify  MOEs)  and 
into  Boxes  4  and  6. 


Although  the  major  MOE  development  has  been  accomplished,  it  will 
still  be  going  on  at  the  more  detailed  levels  of  analysis,  with  the 
inevitable  requirement  to  modify  MOEs  as  more  is  learned  about  the 
system  and  scope  ot  the  analysis. 
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SUMMARY 


This  report  illustrates  that  the  MOEs  associated  with  any  par¬ 
ticular  system  or  analysis  are  hierarchical  in  nature.  The  top-level 
hierarchy  included  factors  such  as  cost,  survivability,  reliability, 
and  capability.  The  second-level  hierarchy  was  used  to  illustrate  an 
expansion  of  capability  MOEs,  which  were  the  topic  of  this  paper. 

In  the  capability  hierarchy,  there  is  a  transition  between  mission 
MOEs  and  individual  task,  performance.  It  is  possible  to  go  down  at 
least  two  levels  in  the  hierarchy  before  encountering  individual 
system  components  or  human  operators  of  components.  There  need  not  be 
a  "visible"  connection  between  mission  capability  and  individual  human 
operator  performance,  unless  special  attention  is  given  to  making  the 
connection  explicit. 


IMPLICATIONS  FOR  HUMAN  FACTORS 

Implications  that  can  be  taken  from  the  material  presented  in  this 
report  are  given  in  the  following  three  categories. 


Human  Factors  Analyst 


An  analyst  working  at  any  level  in  the  MOE  hierarchy  must  be 
familiar  with  the  MOEs  at  the  levels  both  above  and  below  his  or  her 
own.  The  analyst  must  understand  the  lower  level  MOEs  (or  performance 
data)  so  as  to  not  misuse  them,  and  the  product  of  the  analysis  must 
be  useful  at  the  level  above.  The  analyst  must  be  able  to  limit  the 
scope  of  the  work  so  that  something  can  be  accomplished,  yet  have  a 
broad  enough  overview  so  that  the  correct  inputs  are  used  from  below, 
and  the  "right"  questions  are  answered  by  the  analysis  results. 


Human  Factors  Experimenter 


The  Human  Factors  experimenter  conducting  simulations  or  tests 
within  a  project  structure  must  understand  what  kind  of  data  is 
required,  and  the  form  of  the  data  that  is  usable  in  the  analysis. 
The  experimenter  must  either  be  given  specific  data  requirements,  or 
know  enough  to  develop  the  requirements  by  examining  the  type  of 
analysis  and  MOEs  being  used. 


The  experimenter  collecting  so-called  "baseline”  data  has  a  more 
difficult  task.  Generic  experiments  have  no  system  development  struc¬ 
ture  from  which  requirements  can  be  derived.  There  are  no  MOEs  that 
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require  input  data.  It  is  often  the  case  that  such  data  are  not 
useful  when  analysis  requirements  finally  come  along. 

One  could  argue  that  such  generic  experiments  are  useful  only  in 
the  human  factors  education  context  (i.e.,  to  allow  the  student  to 
demonstrate  the  ability  to  design  and  conduct  experiments). 

Such  experimental  results  might  be  made  more  useful  if  a  system 
development  program  is  hypothesized  and  MOEs  are  developed  as  a  first 
step  in  defining  the  experiment.  The  inclusion  of  this  step  in  the 
design  of  generic,  or  "baseline  data"  experiments  is  worth  trying  in 
order  to  increase  the  usefulness  of  the  results. 


Huaan  Factors  Manager 

This  report  has  provided  some  insight,  or  another  view,  of  why  a 
"disconnect"  between  operator  performance  (or  human  factors)  and 
system  effectiveness  is  often  perceived.  The  problem  in  establishing 
the  worth  of  human  factors  in  the  system  design  process  lies  partly  in 
the  fact  that  operator  performance  is  found  at  the  lower  levels  in  the 
MOE  hierarchy.  The  program  manager,  sponsor,  or  person  with  the  money 
tends  to  be  concerned  about  the  higher  level  MOEs.  Operator  perfor¬ 
mance  is  not  often  seen  explicitly  at  these  levels. 

Operator  performance,  per  se,  must  be  carried  upward  in  the 
analysis  procedure  to  demonstrate  the  payoff  in  adequately  funding  the 
human  factors  part  of  a  development  effort.  The  hypothetical  MOEs  for 
baseline  work  referred  to  above  would  also  show  human  factors  worth. 
The  human  factors  analyst  or  engineer  may  be  the  only  one  motivated 
and  qualified  to  conduct  such  analysis. 


MOE  DEVELOPMENT  PROCEDURES 

As  mentioned  earlier  in  the  report,  the  guidelines  for  developing 
MOEs  are  collected  together  here  in  the  summary  for  easy  reference  and 
use.  The  practitioner  can  follow  the  guidelines  and  go  back  to  the 
text  when  clarification  is  required. 

It  is  common  to  have  more  than  one  mission  objective,  and  it  is 
important  to  address  them  separately  in  the  analysis.  Figures  8  and  y 
show  how  to  separate  the  objectives.  Figure  10  and  Table  18  show  the 
procedures  to  follow  in  developing  capability  top-level  MOEs.  Figures 
11  and  12  can  be  used  to  generate  lower  level  MOEs. 

At  this  point  in  the  analysis  procedures,  the  analyst  has  moved 
beyond  Box  3  in  Figure  1,  and  has  completed  the  subject  of  this 
report . 
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FIGURE  9.  Second  Step  in  Separating  Mission  Objectives. 
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TABLE  18.  Steps  to  Follow  in  Thinking  up  Candidates 
for  Top-Level  Measures  of  Effectiveness. 


1.  List  the  single  mission  objective  as  determined  in  the  procedure 
illustrated  in  Figure  10. 

2.  List  questions  that  need  answering  as  obtained  from  sponsors, 
program  managers,  design  engineers,  etc. 

3.  Develop  a  list  of  many  conceivable  MOEs  for  the  mission  objective, 
without  any  initial  constraints.  Use  a  brainstorming  technique  if 
possible. 


4.  Write  a  description  of  the  Quantification  Concepts  for  each  MOE 
listed. 

5.  Categorize  the  MOEs  into  groups  of  similar  measures. 

6.  Reduce  the  list  by  combining  (or  discarding)  duplicate  or 
redundant  MOEs. 

7.  Further  eliminate  MOEs  from  the  list  beacause  of: 

a.  Technical  infeasibility 

b.  Economnic  infeasibility 

c.  Weak  or  no  relation  to  mission  objective 

8.  List  the  remaining  Top-Level  MOEs  for  the  Single  Mission  Objective 
given  in  (1)  above. 

9.  Repeat  the  process  for  each  Single  Mission  Objective. 
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Appendix 

PUBLISHED  MOEs  AND  PERFORMANCE  MEASURES 


In  addition  to  the  general  guidelines  cited  and  provided  in  this 
report,  some  of  the  documents  reviewed  give  examples  of  specific  MOEs 
and  performance  measures  (Table  A-l).  The  procedures  recommended  in 
this  report  should  still  be  followed  to  ensure  continuity  within  the 
MOE  hierarchy.  But  information  given  in  Table  A-l  can  be  used  in  the 
brainstorming  process  referred  to  in  Table  18. 


TABLE  A-l.  Examples  of  Specific  MOEs. 


Subject 


Reference 


Army  weapons  and  tactics 

3,  11,  13 

Camouflage 

14 

Navy  carrier  landings 

7,  8 

Naval  operations  (Top-Level) 

13 

Air-to-air  combat 

9,  16 

Airborne  forward  air  controller 

n 

Air-to-surf ace  attack 

12,  16 

Shipboard  nuclear  security 

17 

Pilot  performance 

18,  19 

i 

Air  reconnaissance 

16 

mmmmm 
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MOE  DEFINITIONS 

In  the  Navy,  availability  is  equivalent  to  readiness  and 
dependability  is  equivalent  to  reliability. 

Another  definition  is  found  in  a  Navy  document  (Reference  20): 

Systems  performance  effectiveness  can  be  defined  as,  "a 
measure  of  t  ,e  extent  to  which  a  system  can  be  expected  to 
complete  its  assigned  mission  within  an  established  time 
frame  under  stated  environmental  conditions". 

An  Army  document  (Reference  13)  gives  the  same  definition  as  has 
been  found  in  Air  Force  sources  (Reference  2).  Reference  13  gives  a 
more  specific  definition  of  the  three  components  of  systems 
effectiveness : 

1.  Availability  is  a  measure  of  the  degree  to  which  an  item  (or 
is  the  chance  that  a  system)  is  in  the  operable  or  committable  state 
at  the  start  of  a  mission  when  the  mission  is  called  for  at  an  unknown 
(random)  point  in  time. 

2.  Dependability  is  a  measure  of  the  system  operating  condition 
during  the  performance  of  the  mission,  given  the  condition  of  the 
system  at  the  start  of  the  mission  (availability).  Reliability, 
survivability,  and  maintainability  are  included  in  the  dependability 
concepts . 

3.  Capability  is  a  measure  of  or  the  chance  that  a  system  will 
achieve  mission  objectives  given  the  conditions  during  the  mission 
(dependability) . 

The  concept  of  breaking  down  system  effectiveness  into  three 
components  was  evidently  originated  in  an  early  Air  Force  contracted 
study  (Reference  21),  and  has  been  found  useful  ever  since. 

A  definition  of  a  MOE  is  a  starting  point  in  its  description.  The 
quantitative  measure  used  to  compare  the  effectiveness  of  alternative 
courses  of  action  in  achieving  the  objective  is  called  the  Measure  of 
Effectiveness  (Reference  22). 

A  criterion  is  a  measure  or  standard  by  which  performance  of  a 
system  is  evaluated  (Reference  23). 

A  measure  of  effectiveness  is  a  quantitative  expression  of  the 
degree  to  which  a  system  fuLtills  its  objectives  (Reference  24). 
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MOE  definitions  found  in  documents  in  the  Armed  Forces  get  a 
little  more  complicated: 


A  FOM  is  a  measure  of  system  effectiveness  pertinent  to 
one  or  more  mission  requirements  (Reference  2). 


Reference  2  breaks  the  concept  into  three  parts:  availability,  depend¬ 
ability,  and  capability.  To  expand  upon  the  above  definition: 


A  capability  FOM  is  a  measure  of  the  ability  of  a  system 
to  achieve  mission  objectives,  given  specific  system  con¬ 
ditions  during  the  mission.  It  specifically  accounts  for 
the  performance  spectrum  of  a  system  (Reference  2). 


Reference  20  uses  several  terms:  figure  of  merit,  system  effectiveness 
figures,  system  performance  effectiveness  measures,  effectiveness 
measures,  and  effectiveness  criteria.  This  measure  is  divided  into 
two  distinct  but  related  factors: 


PC 

PT 


the  performance  variable, 
associated  range  of  values. 


or  capability  and  its 


the  detailed  time-dependency  of  performance,  which  allows 
reliability,  maintainability,  availability,  etc.,  to  be 
treated  as  modifiers  of  performance. 


MOE  HIERARCHY 


Table  A-l  (page  47)  shows  what  could  be  called  a  hierarchy  of 
analysis;  each  level  may  have  different  analysis  techniques  and 
different  MOEs.  The  concept  of  hierarchies  will  be  expanded  here  to 
help  produce  the  guidelines  for  developing  MOEs. 


Tables  A-2  through  A-5  show  listings  similar  to  that  shown  in 
Table  A-l.  Although  the  references  from  which  these  tables  were  taken 
were  written  for  different  purposes,  the  common  concept  is  that  there 
are  measures  of  effectiveness  (or  performance)  at  each  of  these 
levels.  They  are  combined  in  some  way  to  get  the  measure  at  the  next 
higher  level. 
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TABLE  A-2.  Figure-of-Merit  Hierarchy  (Kererence  2). 


FOM  level 

Example 

Top-level 

! 

i 

Force  structure  composed  of  different 
system  classes. 

First-level 

j 

Single  system  with  "dependent”  system 
(e.g.,  aircraft  equipped  with  missile.) 

Second-level 

Single  system 

Third-level 

Subsystem  or  human  operator 

1 

TABLE 

A-3.  Mission  Hierarchy. 

(Reference  16) 

Mixed  system  force  missions 
Multiple  system  missions 
Single  system  missions 

TABLE  A-4.  Assessment  Hierarchy. 
(Reference  14) 

Force  level 

Tactical  operations 

System  operations 


Design 


TABLE  A-5.  The  Hierarchy  of  Measures 
of  Effectiveness  (Reference  11). 

National  security  (Political  level) 

Major  mission  (OSD-Joint  Chiefs  of  Staff  level) 

Program  element  (Department  of  the  Army  level) 

Military  worth/combat  effectiveness 

System  performance  capability  level 

System  characteristics  level 


Component  characteristics  level 


HOE  CHARACTERISTICS 


It  may  be  helpful  to  cite  some  of  the  properties  of  MOEs  that 
analysts  have  found  necessary,  useful,  and/or  desirable. 

Multiplicity 

The  difficulty  of  finding  a  single  criterion  by  which  to  judge  a 
system  has  been  a  common  experience  (References  3,  14,  25,  and  26). 
When  a  system  is  tasked  to  perform  several  independent  functions 
(e.g.,  navigation  and  weapon  delivery),  it  is  often  advisable  to  carry 
the  separate  MOEs  along  as  far  as  possible  up  the  hierarchy.  A 
requirement  on  the  system  to  perform  different  assignments  of  missions 
(e.g.,  target  attack  and  search  and  rescue)  also  leads  to  using 
separate  MOEs. 

Some  criteria  used  in  equipment  design  cannot  easily  be  quanti¬ 
tatively  related  to  system  performance,  yet  they  "make  good  sense". 
The  comfort,  long  term  health  (e.g.,  avoid  hearing  impairment),  and 
actual  performance  by  the  human  operator  on  a  mission  all  have 
separate  MOEs  that  should  affect  design  decisions. 

Multiple  MOEs  must  be  combined  into  one  MOE  in  some  fashion  for 
use  in  each  design  decision  that  is  made.  But  for  most  levels  of 
technical  analysis,  multiple  MOEs  are  the  rule  rather  than  the 
exception. 
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Simplicity 

To  offset  the  notion  of  several  MOEs,  Attaway  advises  that  a  good 
general  guideline  in  selecting  a  MOE  is  "to  choose  the  narrowest  goal 
possible  in  order  to  minimize  analytic  effort"  (Reference  26).  The 
simpl i t icat ion  of  MOEs  during  tbe  analytic  process,  in  order  to  end  up 
with  something  manageable,  is  also  a  common  experience  (References  1 
and  27 ) . 


Quantifiable/Measurable 

Although  design  and  other  types  of  decisions  are  made  in  the 

course  of  a  program  based  on  information  that  is  qualitative,  an  oft- 
cited,  required  characteristic  of  MOEs  is  that  they  be  quantitative 
(References  11,  20,  22,  and  28)  or  probabilistic  (Reference  28).  The 
concept  of  describing  capability  in  terms  of  probabilities,  percents, 
or  ratios  of  numbers  could  be  included  under  the  quantitative 
descriptor. 

Some  documents  (References  13,  22,  and  29)  have  also  included  the 
requirement  that  a  MOE  be  measurable.  If  one  interpreted  that 
requirement  as  meaning  measurable  in  theory,  or  in  concept,  most  MOEs 
that  have  been  used  would  qualify.  Lower  level  MOEs,  such  as  com¬ 
ponent  or  human  operator  performance  (Figure  3)  can  and  are  measured 

routinely.  MOEs  used  in  test  and  evaluation  must  be,  and  are 

measured.  However,  it  is  seldom  practical  to  instrument  and  measure 
higher  level  MOEs  such  as  the  outcome  of  a  battle.  Many  MOEs  used  in 
effectiveness  analysis  are  never  measured  and,  indeed,  it  would  be 
impractical  to  do  so  (Reference  l). 


Mission  Related 

The  measure  of  effectiveness  evaluates  or  predicts  aspects  of 
system  performance  relevant  to  operational  issues.  It  therefore 
should  be  operationally  credible  (Reference  28)  or  in  operationally- 
oriented  form  that  can  readily  understood  and  used  in  planning 
(Reference  21). 


Meaningful/Useful 

As  l^uade  wrote  some  time  ago,  "working  out  a  systems  analysis  with 
a  bad  criterion  is  equivalent  to  answering  the  wrong  question" 
(Reference  25).  Others  discussing  the  characteristics  of  MOEs  have 
expressed  the  same  requirement. 


MOEs  should  be: 


1.  oriented  to  issues,  to  the  right  decision  level  and  the 
decision-maker  (Reference  30) 

2.  Relevant  (Reference  13) 

3.  Meaningful  to  the  system  designer,  system  analyst,  user, 
mission  analyst  (Reference  20) 

4.  Specifically  related  to  the  design  item  characteristics 
(Reference  14) 

The  above  statements  illustrate  the  many  uses  of  effectiveness 
analysis  and  the  importance  in  choosing  proper  MOEs.  MOEs  should  be 
multiple,  simple,  quantitative,  mission-related,  system-related,  and 
meaningful  to  a  number  of  people. 
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1  Army  Missile  Command,  Redstone  Arsenal  (DRXHE-MI) 

1  Army  Missile  Command,  Redstone  Scientific  Information  Center,  Redstone  Arsenal  (DRSMI-RPRD) 

2  Army  Mobility  Equipment  Research  and  Development  Command.  Fort  Belvoir 

DRDME-RT,  Camouflage  Laboratory  (1) 

Library  (1) 

1  Army  Training  and  Doctrine  Command,  Fort  Monroe  (Technical  Library) 

1  Army  Aeromedical  Research  Laboratory  .  Fort  Rucker  (Technical  Library) 

1  Army  Ballistic  Research  Laboratory,  Aberdeen  Proving  Ground  (DRDAR-TSB-S  (ST1NFO)) 

2  Army  Human  Engineering  Laboratory,  Aberdeen  Proving  Ground 

4  Army  Materiel  Systems  Analysis  Activity,  Aberdeen  Proving  Ground 

DRXSY-D  (1) 

DRXSY-C4cDS  (1) 

DRSXY-J  (1) 

Technical  Library  (1) 

1  Fort  Huachuca  Headquarters.  Fort  Huachuca  (Technical  Library) 

1  Night  Vision  Laboratory,  Fort  Belvoir  (Technical  Library) 

1  Office  Chief  of  Research  and  Development 

2  White  Sands  Missile  Range 

TRASANA/ATOR-D  (1) 

Technical  Library  (1) 

1  Air  Force  Logistics  Command,  Wright -Patterson  Air  Force  Base 
1  Air  Force  Systems  Command,  Andrews  Air  Force  Base  (SDZ) 

1  Tactical  Air  Command,  Langley  Air  Force  Base  (Technical  Library) 

1  Aeronautical  Systems  Division,  Wright-Patterson  Air  Force  Base  (ASD/AERS) 

5  Air  Force  Armament  Division,  Eglin  Air  Force  Base 

AD/XRSS  (3) 

AFATL/DLYW  (2) 

1  Air  Force  Intelligence  Service,  Bolling  Air  Force  Base  (AFIS/INTAW,  Maj.  R,  Lecldider) 

2  Air  Force  Medical  Research  Laboratory,  Wright-Patterson  Air  Force  Base 

AFAMRL/HEA,  Dr,  T.  Furness  (1) 

AFAMRL/HEF,  Dr.  Topmiller  (1) 

1  Air  Force  Wright  Aeronautical  Laboratories.  Wright-Patterson  Air  Force  Base  (AFWAL/FGR,  Dr.  J. 

1  Pacific  Air  Forces  (Headquarters,  PACAF/OA  (Operations  Analysis)) 

1  Defense  Intelligence  Agency  (Technical  Library) 

3  Under  Secretary  of  Defense  for  Research  and  Engineering  (OAD)  E4cLS  (Capt.  Chatelier) 

12  Defense  Technical  Information  Center,  Alexandria 

1  Calspan  Corporation,  Advanced  Technology  Center,  Buffalo,  NY  (Life  Sciences  Avionics  Department) 

2  California  State  University,  Northridge,  CA 

Psychology  Department 
Dr.  Sanders  (1) 

Dr,  Streimer  (1) 

2  Hudson  Institute,  Incorporated,  Center  for  Naval  Analyses,  Alexandria,  VA 
G.  L.  Richardson  (1) 

Technical  Library  (1) 


Reising) 
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1  ( General  Research  Corporation,  Santa  Barbara.  C.A 

2  Honeywell.  Incorporated.  Systems  and  Research  Center.  Minnepolis.  MN 

Dr.  L.  Miller  (1) 

Dr  J.  Wald  (1) 

1  Hughes  Aircraft  Company,  Los  Angeles.  CA  (Display  Systems  Department) 

I  Homan  Factors  Research.  Goleta,  CA  (C-320) 

1  IBM.  Owego.  NY  (Human  Factors  Group,  304A535) 

3  Institute  for  Defense  Analyses  (IDA),  Alexandria,  VA 

Dr.  J.  Orlansky  (2) 

Technical  Library  (1) 

2  LTV  Aerospace  and  Defense,  Systems  Division.  Dallas.  TX  (Human  Factors  Group) 

2  Martin-Marietta  Aerospace,  Orlando.  FL 

Image  Processing  Laboratory  (MS  362)  (l) 

Technical  Library  (1) 

2  McDonnell  Douglas  Corporation,  Long  Beach,  CA  (Director.  Scientific  Research,  R&D  Aircraft  Division) 

3  McDonnell  Douglas  Corporation,  St.  Louis,  MO 

Engineering  Psychologs.  Department  E422  (2) 

Technical  Library  (1) 

1  National  Academy  of  Sciences  (Vision  Committee) 

3  Northrop  Corporation.  Aircraft  Division,  Hawthorne,  CA 

Avionics  Systems  Definitions  Group  (3511) 

Human  Factors  Department  (1) 

Technical  Library  (1) 

1  Perceptronics.  Incorporated,  Woodland  Hills.  CA 

2  Rand  Corporation,  Santa  Monica.  CA 

Natalie  E.  Crawford  (1) 

Technical  Library  (1) 

2  Rockwell  International  Corporation,  Autonetics  Group,  Anaheim.  CA  (Human  Factors  Group) 

1  Rockwell  International  Corporation,  Duluth,  GA  (Technical  Library) 

4  The  Boeing  Company.  Seattle,  WA  (Crew  Systems,  MS  41-08) 

l  The  Johns  Hopkins  University,  Applied  Physics  Laboratory.  Laurel,  MD  (Technical  Library) 

1  University  of  Southern  California,  Los  Angeles.  CA  (Dr.  DeCreene.  Institute  of  Safety  and  Systems  Management) 

2  Virginia  Polytechnic  Institute  and  State  University.  Blacksburg.  VA  (Industrial  Engineering  Department) 


